Human trust calibration for autonomous driving agent of vehicle

ABSTRACT

An autonomous driving agent is provided. The autonomous driving agent determines a set of observations from sensor information of a sensor system of a vehicle. The set of observations includes human attention information for a scene of surrounding environment and a level of human reliance as indicated by human inputs to the autonomous driving agent. The autonomous driving agent estimates, based on the set of observations, belief states for a first state of human trust on the autonomous driving agent and a second state of human&#39;s cognitive workload during journey. The autonomous driving agent selects, based on the estimated belief states, a first value for a first action associated with a level of automation transparency between a human user and the autonomous driving agent and controls a display system based on the selected first value to display a cue for calibration of the human trust on the autonomous driving agent.

BACKGROUND

With advancements in self-driving technology, there has been a rise inadoption of sensor-based driving automation systems in vehicles. Theself-driving technology may be implemented through driving automationsystems, such as Advanced Driver-Assistance Systems (ADAS). Human usersare increasingly becoming dependent on ADAS features, such as adaptivecruise control, lane assist, or collision avoidance. Despite significantadvancements, human supervision and intervention may still be required.It has also been found that human trust may play a critical role ininteractions between a human user of the vehicle and a drivingautomation system of the vehicle. For example, low levels of human trustmay lead to disuse of the driving automation system. Whereas, anexcessively reliance on the capabilities of the driving automationsystem under unsafe conditions or situations outside of the scope ofautomation design, may lead to over trust which may result in unintendedconsequences for the human user.

Limitations and disadvantages of conventional and traditional approacheswill become apparent to one of skill in the art, through comparison ofdescribed systems with some aspects of the present disclosure, as setforth in the remainder of the present disclosure and with reference tothe drawings.

SUMMARY

An exemplary aspect of the disclosure provides an autonomous drivingagent of a vehicle. The autonomous driving agent includes circuitrycoupled to a display system of the vehicle. The circuitry may determinea set of observations from sensor information acquired via at least onesensor of the vehicle. The set of observations may include humanattention information associated with a scene of a surroundingenvironment of the vehicle and a level of human reliance as indicated byhuman inputs to the autonomous driving agent. The circuitry mayestimate, based on the determined set of observations, a set of beliefstates for a first state of human trust on the autonomous driving agentand a second state of a human's cognitive workload in course of ajourney. Further, the circuitry may select, based on the estimated setof belief states, a first value of a set of values for a first actionassociated with a level of automation transparency between a human userof the vehicle and the autonomous driving agent. Based on the selectedfirst value, the circuitry may control the display system to display acue for calibration of human trust on the autonomous driving agent.

Another exemplary aspect of the disclosure provides a method for humantrust calibration for an autonomous driving agent of a vehicle. Themethod may be implemented by any computing system, such as by anautonomous driving agent for a vehicle. The method may includedetermining a set of observations from sensor information acquired viaat least one sensor of the vehicle. The set of observations may includehuman attention information associated with a scene of a surroundingenvironment of the vehicle and a level of human reliance as indicated byhuman inputs to the autonomous driving agent. The method may furtherinclude estimating, based on the determined set of observations, a setof belief states for a first state of human trust on the autonomousdriving agent and a second state of a human's cognitive workload incourse of a journey. The method may further include selecting, based onthe estimated set of belief states, a first value of a set of values fora first action associated with a level of automation transparencybetween a human user of the vehicle and the autonomous driving agent.The method may further include controlling, based on the selected firstvalue, a display system to display a cue for a calibration of the humantrust on the autonomous driving agent.

Another exemplary aspect of the disclosure provides a non-transitorycomputer-readable medium having stored thereon computer implementedinstructions that, when executed by an autonomous driving agent of avehicle, causes the autonomous driving agent to execute operations. Suchoperations may include determining a set of observations from sensorinformation acquired via at least one sensor of the vehicle. The set ofobservations may include human attention information associated with ascene of a surrounding environment of the vehicle and a level of humanreliance as indicated by human inputs to the autonomous driving agent.The operations may further include estimating, based on the determinedset of observations, a set of belief states for a first state of humantrust on the autonomous driving agent and a second state of a human'scognitive workload in course of a journey. Further, the operations mayinclude selecting, based on the estimated set of belief states, a firstvalue of a set of values for a first action associated with a level ofautomation transparency between a human user of the vehicle and theautonomous driving agent. Thereafter, based on the selected first value,the operations may include controlling a display system to display a cuefor a calibration of the human trust on the autonomous driving agent.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary environment for calibration of humantrust on an autonomous driving agent of a vehicle, in accordance with anembodiment of the disclosure.

FIG. 2 is a block diagram of the autonomous driving agent of FIG. 1, inaccordance with an embodiment of the disclosure.

FIG. 3 is a block diagram of an exemplary vehicle that implements theautonomous driving agent of FIG. 1, in accordance with an embodiment ofthe disclosure.

FIG. 4 is a diagram that illustrates an exemplary trust-workload modelfor calibration of human trust on the autonomous driving agent of FIG.1, in accordance with an embodiment of the disclosure.

FIG. 5 is a diagram that illustrates exemplary operations for trainingthe exemplary trust-workload model of FIG. 4, in accordance with anembodiment of the disclosure.

FIG. 6 is a diagram that illustrates exemplary operations for humantrust calibration for the autonomous driving agent of FIG. 1, inaccordance with an embodiment of the disclosure.

FIG. 7A is a diagram that illustrates an exemplary scenario forcalibration of human trust on the autonomous driving agent of FIG. 1 incourse of a journey, in accordance with an embodiment of the disclosure.

FIG. 7B is a diagram that illustrates an exemplary scenario forcalibration of human trust on the autonomous driving agent of FIG. 1 incourse of a journey, in accordance with an embodiment of the disclosure.

FIG. 7C is a diagram that illustrates an exemplary scenario forcalibration of human trust on the autonomous driving agent of FIG. 1 incourse of a journey, in accordance with an embodiment of the disclosure.

FIG. 8 is a flowchart that illustrates an exemplary method forcalibration of human trust on the autonomous driving agent of FIG. 1, inaccordance with an embodiment of the disclosure.

The foregoing summary, as well as the following detailed description ofthe present disclosure, is better understood when read in conjunctionwith the appended drawings. For the purpose of illustrating the presentdisclosure, exemplary constructions of the preferred embodiment areshown in the drawings. However, the present disclosure is not limited tothe specific methods and structures disclosed herein. The description ofa method step or a structure referenced by a numeral in a drawing isapplicable to the description of that method step or structure shown bythat same numeral in any subsequent drawing herein.

DETAILED DESCRIPTION

The following described implementations may be found in a disclosedautonomous driving agent for a vehicle. The disclosed autonomous drivingagent may operate on a paradigm that may anticipate human interaction orhuman behavior and may influence a human user to make optimal choicesabout use of the autonomous driving agent when the vehicle is in courseof a journey and is controlled by the autonomous driving agent. Thehuman interaction or the human behavior may be associated with humantrust in automation (i.e. autonomous driving agent) in real time or nearreal time. For such an approach, the present disclosure provides aquantitative method to predict the human behavior and to determineoptimal interventions to influence the human behavior. As the humantrust and human's cognitive workload may be coupled to each other andmay influence the use of the autonomous driving agent 102, theautonomous driving agent implements a trust-workload model as per aPartially Observable Markov Decision Process (POMDP) framework to modela trust-workload relationship of the human behavior in an autonomousdriving context (such as a hands-off SAE (Society of AutomotiveEngineers) level 2 driving context). In such a context, thetrust-workload model may provide the ability to measure states of thehuman trust and the human's cognitive workload, continuously, and inreal time or near real time in course of a journey, using belief stateestimates.

In order to calibrate the human trust, the autonomous driving agent maydefine a reward function as a function of the state of the human trustand the automation reliability. The trust-workload model may considerthe automation reliability, the automation transparency, and the scenecomplexity, together with human reliance on the autonomous driving agentand human attention (e.g., in terms of eye gaze behavior in course of ajourney), to model human trust-workload relationship. The autonomousdriving agent may implement an optimal control policy that may be usedto dynamically vary the automation transparency based on the beliefstate estimates of the trust-workload model for the states of the humantrust and the human's cognitive workload. Specifically, the autonomousdriving agent may display cue(s) on a display system of the vehicle tosetup a particular level of the automation transparency between thehuman user of the vehicle and the autonomous driving agent. For example,such cue(s) may include visual indicators for objects (e.g., pedestrian,vehicles, road, etc.) detected in a driving scene of a surroundingenvironment of the vehicle or decisions (e.g., left turn, right turn,etc.) made by the autonomous driving agent. By displaying such cue(s),the autonomous driving agent may be able to showcase one or more of:intended actions, actions that it performs, plans, reasons, or itsunderstanding of the scene complexity of the surrounding environment. Bydoing so, the autonomous driving agent may be able to provide a feedbackto the human user to calibrate the human trust on the autonomous drivingagent. This may be needed especially if the human user is determined tobe in a state of over trust or a state of under trust for the autonomousdriving agent in relation to various factors, such as the automationreliability.

While representation of cues may increase the human trust on theautonomous driving agent, however, too many cues may distract the humanuser and may lead to an increase in the human's cognitive workload,especially if the human's cognitive workload is already high. Thus, theautonomous driving agent may display an appropriate number and types ofcues according to the optimal control policy for the automationtransparency and the belief state estimates for the states of the humantrust and the human's cognitive workload.

FIG. 1 illustrates an exemplary environment for calibration of humantrust on an autonomous driving agent of a vehicle, in accordance with anembodiment of the disclosure. With reference to FIG. 1, there is shownan exemplary network environment 100. In the exemplary networkenvironment 100, there is shown an autonomous driving agent 102 and avehicle 104 that may include the autonomous driving agent 102. There isfurther shown a sensor system 106, a server 108, and a communicationnetwork 110 which may be established among one or more of: theautonomous driving agent 102, the vehicle 104, the sensor system 106,and the server 108. There is further shown a view 112 of a surroundingenvironment 114 and a side view 116 of the vehicle 104 in thesurrounding environment 114. The surrounding environment 114 may includethe vehicle 104 and other objects, such as pedestrians, trees, roads,street signs, other vehicles, or traffic lights. In the side view 116,there is shown a display system 118 and a human user 120 who is shown tobe seated on a driver's seating position 122 in the vehicle 104. In theside view 116, there is further shown an exemplary implementation of thesensor system 106 through a gaze detector 106 a and a front facingcamera 106 b.

The autonomous driving agent 102 may include suitable logic, circuitry,interfaces, and/or code that may be configured to operate the vehicle104 based on a level of automation, as defined by Society of AutomotiveEngineers (SAE). The autonomous driving agent 102 may be an exemplaryimplementation of an action automation, where the autonomous drivingagent 102 may take an action, unless intervened upon by the human user120. Unlike a decision automation, where the interaction of the humanuser 120 with an autonomous agent may be characterized by a compliancewhenever the autonomous agent presents a recommendation, the actionautomation may be characterized by the human user 120 continuouslyeither relying on the autonomous driving agent 102 or intervening totake over control of the autonomous driving agent 102.

The autonomous driving agent 102 may be responsible for human trustcalibration when the vehicle 104 is in course of a journey. For example,the autonomous driving agent 102 may determine whether to display a cue(for example, a bounding box or another visual identifier) over a visualrepresentation of a scene of the surrounding environment 114 displayedon the display system 118. By displaying the cue, the autonomous drivingagent 102 may be able to showcase its understanding of scene complexity(including objects) of the surrounding environment 114. By doing so, theautonomous driving agent 102 may be able to provide a feedback to thehuman user 120 to calibrate the human trust on the autonomous drivingagent 102. This may be needed especially if the human user 120 isdetermined to be in a state of over trust or a state of under trust forthe autonomous driving agent 102 in relation to various factors, such asa level of automation reliability.

In an exemplary implementation, the autonomous driving agent 102 may beimplemented as a computer, such as an in-vehicle Electronic Control Unit(ECU), onboard the vehicle 104. In such a case, the autonomous drivingagent 102 may be implemented as a specialized electronic circuitry thatmay include one or more processors to control different functions, suchas, but not limited to, engine operations, steering controls,transmission controls, braking controls, communication operations, ordata acquisition from the sensor system 106.

The autonomous driving agent 102 may be configured to communicate withdifferent in-vehicle components, such as a vehicle control system, anin-vehicle infotainment (IVI) system, an in-car entertainment (ICE)system, an automotive Head-up Display (HUD), an automotive dashboard, asmartphone, a human-machine interface (HMI), a computer workstation, ahandheld computer, a portable consumer electronic (CE) device, a server,or other computing devices.

Although, in FIG. 1, the autonomous driving agent 102 is shown as a partof the vehicle 104; however, the disclosure may not be so limiting andin some embodiments, the autonomous driving agent 102 may be a separateentity, which may control the vehicle 104 remotely via the communicationnetwork 110, without a deviation from the scope of the disclosure. Insuch an implementation of the autonomous driving agent 102, theautonomous driving agent 102 may be implemented on one of: aVehicle-to-Everything (V2X) network, an application server, a webserver, a cloud server (or a cluster of cloud servers), a factoryserver, a consumer-electronic (CE) device, or a dedicated vehiclecontrol server.

The vehicle 104 may be a self-driving vehicle and the autonomous drivingagent 102 may be configured to operate the self-driving vehicle based ona level of automation, as defined by the SAE. For example, theautonomous driving agent 102 may operate the vehicle 104 based on a SAElevel-2 vehicle automation, in which the steering and brake/accelerationof the vehicle 104 may be controlled by the autonomous driving agent102. Examples of the vehicle 104 may include, but are not limited to, atwo-wheeler vehicle, a three-wheeler vehicle, a four-wheeler vehicle, ahybrid vehicle, or a vehicle with autonomous drive capability that usesone or more distinct renewable or non-renewable power sources. A vehiclethat uses renewable or non-renewable power sources may include a fossilfuel-based vehicle, an electric propulsion-based vehicle, a hydrogenfuel-based vehicle, a solar-powered vehicle, and/or a vehicle powered byother forms of alternative energy sources.

In at least one embodiment, the vehicle 104 may include a capability todrive itself from a starting point to a destination point based onvarious vehicle-related technologies and sensors, including adaptivecruise control, active steering, anti-lock braking systems (brake bywire), Global Navigation Satellite System (GNSS) navigation technology,lasers, cameras, RADAR system(s), On-Board Units (OBUs), or LightDetection and Ranging (LIDAR) system(s). Details of such vehicle-relatedtechnologies and sensors are omitted from the disclosure for the sake ofbrevity.

Although, in FIG. 1, the vehicle 104 is illustrated as a four-wheelercar; however, the present disclosure may not be limited to theimplementation of the vehicle 104 as a four wheeler vehicle. In at leastone embodiment, the vehicle 104 may be one of: an Unmanned AerialVehicle (UAV), a manned self-flying vehicle (such as an airplane thathas the capability to fly using an autopilot system), a waterbornevehicle (such as a submarine or a ship), an industrial robot (e.g., anarticulated robot, a SCARA robot, a delta robot, and a cartesiancoordinate robot), an agricultural robot, a mobile robot (e.g., awarehouse robot, an Automated Guided Vehicle (AGV), or an AutonomousMobile Robots (AMR)), a telerobot, or a service robot. In suchimplementations of the vehicle 104, the human user 120 may be a humanoperator or a human supervisor who may monitor operations of theautonomous driving agent 102 for the vehicle 104.

The sensor system 106 may be a heterogeneous sensor system which mayinclude one or more of: the gaze detector 106 a, the front facing camera106 b, or an event logger 106 c. The gaze detector 106 a and the frontfacing camera 106 b may be configured to be mounted on defined locationson the vehicle 104. The sensor system 106 may be configured to acquiresensor information which may include, for example, scene informationassociated with the surrounding environment 114 and human behavioraldata associated with the vehicle 104 and the surrounding environment114. In at least one embodiment, the acquired sensor information may bemultimodal sensor information of the surrounding environment 114. Theacquisition of such multimodal information may include, for example,gaze detection results from the gaze detector 106 a, a sequence of imageframes from the front facing camera 106 b, and a log of eventsassociated with human inputs to the autonomous driving agent 102 fromthe event logger 106 c.

The gaze detector 106 a may be an in-vehicle sensor with a field-of-view(FOV) which covers at least the passenger compartment of the vehicle104. As shown, for example, the gaze detector 106 a is installed at thecenter of a bottom portion of a windshield of the vehicle 104. Thepresent disclosure may be also applicable to other positions of the gazedetector 106 a, without a deviation from the scope of the disclosure.Examples of the gaze detector 106 a may include, but are not limited to,an infrared camera, a color camera, a depth sensor, or an RGB-Depth (D)sensor.

The front facing camera 106 b may be configured to capture images from ascene in front of the vehicle 104. As shown, for example, the frontfacing camera 106 b may be installed on a top portion of the body of thevehicle 104. In at least one embodiment, there may be other camera unitson the vehicle 104, which may capture a plurality of images frames ofthe surrounding environment 114 from a plurality of viewpoints.

The front facing camera 106 b may include at least one imaging unit, forexample, an imaging sensor, a depth sensor, a Red-Green-Blue (RGB)sensor, and/or an infrared (IR) sensor. Examples of the front facingcamera 106 b may include, but are not limited to, a short-range digitalcamera, a long-range digital camera, a 360-degree camera, anomnidirectional camera, a panoramic camera, an action camera, awide-angle camera, a camcorder, a night-vision camera, a camera with aTime-of-flight (ToF) sensor, and/or other devices with image capturingcapability.

The event logger 106 c may include suitable logic, circuitry,interfaces, and/or code that may be configured to log informationassociated with the autonomous driving agent 102. In an embodiment, theevent logger 106 c may log the human inputs to the autonomous drivingagent 102 and may store the logged human inputs and supplementaryinformation (e.g., timestamp) related to the such human inputs in memory(as shown in FIG. 2 or FIG. 3, for example) associated with theautonomous driving agent 102. The human inputs may include a takeover ofone or more controls of the vehicle 104, such as, but not limited to, asteering control, a braking action, a control related to manualtransmission, or an acceleration control of the vehicle 104 by the humanuser 120. Additionally, or alternatively, the event logger 106 c mayalso log events associated with a control of the vehicle 104 by theautonomous driving agent 102. Examples of an implementation of the eventlogger 106 c may include, but are not limited to, an automotive eventdata recorder (EDR) or a computer-executable event monitoring programthat may be configured to execute on a computer on-board the vehicle104.

The server 108 may include suitable logic, circuitry, interfaces, and/orthat may be configured to generate and train a stochastic model, whichwhen deployed on the vehicle 104, may be used for calibration of thehuman trust on the autonomous driving agent 102. The server 108 mayshare the trained stochastic model with the autonomous driving agent 102and may be responsible for deployment of the trained stochastic model onthe vehicle 104. In some embodiments, the server 108 may be implementedas a cloud server, which may be utilized to execute various operationsthrough web applications, cloud applications, HTTP requests, filetransfer, and the like. Examples of the server 108 may include, but arenot limited to, an application server, a cloud server, a web server, adatabase server, a file server, a mainframe server, or a combinationthereof.

The communication network 110 may include a communication medium throughwhich the autonomous driving agent 102, the vehicle 104, the sensorsystem 106, the server 108, and the display system 118 may communicatewith each other. The communication network 110 may be one of a wiredconnection or a wireless connection Examples of the communicationnetwork 110 may include, but are not limited to, the Internet, a cloudnetwork, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network(PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN).Various devices in the exemplary network environment 100 may beconfigured to connect to the communication network 110 in accordancewith various wired and wireless communication protocols. Examples ofsuch wired and wireless communication protocols may include, but are notlimited to, at least one of a Transmission Control Protocol and InternetProtocol (TCP/IP), User Datagram Protocol (UDP), Hypertext TransferProtocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g,multi-hop communication, wireless access point (AP), device to devicecommunication, cellular communication protocols, and Bluetooth (BT)communication protocols.

The display system 118 may include suitable logic, circuitry, andinterfaces that may be configured to manage display of information fordifferent operational parameters of the vehicle 104. Such informationmay include for example, image frames or a video of the scene, annotatedwith cues to highlight objects, such as nearby vehicles, road-blockages,pedestrians, traffic signs, traffic lights, buildings, or a road portionassociated with scene of the surrounding environment 114. Additionally,such information may include mode information related to various drivingmodes of the vehicle 104, speed-related information, engine speed (i.e.a digital tachometer), Advanced Driver-Assistance System (ADAS) relatedinformation, or fuel information. The display system 118 may be alsoresponsible for animation or transition effects with changes in thedisplayed information over time and based on human inputs. Examples ofthe display system 118 may include, but are not limited to, aMulti-Information Display (MID), an automotive Head-Up Display (HUD), aninstrument cluster, an in-vehicle infotainment system, a navigationsystem, or an Internet-enabled communication device.

The display system 118 may include a display device 118 a which may berealized through several known technologies such as, but not limited to,at least one of a Liquid Crystal Display (LCD) display, a Light EmittingDiode (LED) display, a plasma display, or an Organic LED (OLED) displaytechnology, or other display devices. In accordance with an embodiment,the display device 118 a may refer to a display of the infotainment headunit, a projection-based display, a see-through display, and/or anelectro-chromic display.

The present disclosure refers to certain terms, such as an agent (theautonomous driving agent 102), observations, actions, and states. Itshould be noted that these terms are known terminologies of a PartiallyObservable Markov Decision Process (POMDP), which may be used to definethe disclosed problem of the human trust-workload calibration for theautonomous driving agent 102. In POMDP, an agent (such as the autonomousdriving agent 102) is an actor whose actions affect a state of theenvironment (i.e. the human user 120) and since the state is partiallyobservable, observations of the environment (i.e. the human user 120)may be needed to form a belief (a probabilistic measure for a state).Actions may be optimally decided based on belief states to calibrate thehuman trust-workload while the vehicle 104 is in course of a journey.

During operation, the autonomous driving agent 102 may receive thesensor information from one or more sensors, such as the gaze detector106 a, the front facing camera 106 b, or the event logger 106 c, of thesensor system 106. The sensor information be multimodal information andmay include scene information associated with the surroundingenvironment 114 and human behavioral data associated with the vehicle104 and the surrounding environment 114. For example, the sceneinformation may include images captured by the front facing camera 106 bof an FOV region that covers a drivable area of the surroundingenvironment 114 up to a certain distance (e.g., 10 meters or more) infront of the vehicle 104. The human behavioral data may include thehuman inputs in the course of the journey, especially when theautonomous driving agent 102 may be performing complex tasks, such as,but not limited to, passing through intersections, changing lanes,taking complex turns, or parking maneuver.

The autonomous driving agent 102 may determine a set of observationsbased on the sensor information. For example, the autonomous drivingagent 102 may receive event logs associated with the human inputs of thehuman user 120 from the event logger 106 c and may receive the pluralityof image frames associated with the scene from the front facing camera106 b. The set of observations may include the human attentioninformation associated with the scene of the surrounding environment 114and a level of human reliance on the autonomous driving agent 102 asindicated by the human inputs to the autonomous driving agent 102. Forexample, the autonomous driving agent 102 may determine the humanattention information from the received plurality of image frames andmay determine the level of human reliance based on the received eventlogs associated with the human inputs. The human attention informationmay include information associated with a gaze of the human user 120 onobject(s) of the scene in FOV of the human user 120.

The autonomous driving agent 102 may estimate, based on the determinedset of observations, a set of belief states for a first state of thehuman trust on the autonomous driving agent 102 and a second state ofhuman's cognitive workload in course of a journey of the vehicle 104. Asthe first state of the human trust and the second state of the human'scognitive workload may be partially observable states of the human user120, the set of belief states may be estimated as a probabilisticmeasure for such partially observable states. The estimation of the setof belief states is provided in detail, for example, in FIG. 5.

The autonomous driving agent 102 may select, based on the estimated setof belief states, a first value of a set of values for a first actionwhich may be associated with a level of automation transparency betweenthe human user 120 of the vehicle 104 and the autonomous driving agent102. Herein, the automation transparency may be defined as a descriptivequality of an interface (such as a UI on the display system 118)pertaining to its abilities to afford an operator's (i.e. the human user120) comprehension about an intelligent agent's (i.e. the autonomousdriving agent 102) intent, performance, future plans, and reasoningprocess.

For example, the set of values may be one of: “0” or “1”, where “0” maybe for an action of the autonomous driving agent 102 to hide the cuefrom the display system 118 and “1” may be for another action of theautonomous driving agent 102 to display the cue on the display system118.

It should be noted that the level of automation transparency may be oneof the actions which the autonomous driving agent 102 may be able tocontrol in the course of the journey for the human trust-workloadcalibration. Other actions which may not be in control of the autonomousdriving agent 102, may include, for example, a scene complexity (such asroad intersections, traffic, etc.) or automation reliability of theautonomous driving agent 102. Based on the selected first value, theautonomous driving agent 102 may control the display system 118 todisplay the cue for the calibration of the human trust of the human user120 on the autonomous driving agent 102. Operations related to thecalibration of human trust or human trust-workload are explained indetail, for example, in FIG. 6 and FIGS. 7A, 7B, and 7C.

FIG. 2 is a block diagram of the autonomous driving agent of FIG. 1, inaccordance with an embodiment of the disclosure. FIG. 2 is explained inconjunction with elements from FIG. 1. With reference to FIG. 2, thereis shown a block diagram 200 of the autonomous driving agent 102. Theautonomous driving agent 102 may include circuitry 202, a memory 204, aninput/output (I/O) interface 206, and a network interface 208. In atleast one embodiment, the autonomous driving agent 102 may also includethe display system 118. The network interface 208 may connect theautonomous driving agent 102 to various electronic components of thevehicle 104 or other networking components outside the vehicle (such asa Dedicate Short-Range Communication Channel (DSRC) Roadside Unit(RSU)), via the communication network 110. A person of ordinary skill inthe art will understand that the autonomous driving agent 102 may alsoinclude other suitable components or systems, in addition to thecomponents or systems illustrated herein to describe and explain thefunction and operation of the present disclosure. A description of suchcomponents or systems is omitted herein for the sake of brevity.

The circuitry 202 may include suitable logic, circuitry, and/orinterfaces that may be configured to execute program instructionsassociated with different operations to be executed by the autonomousdriving agent 102. For example, some of the operations may includereception of the sensor information, determination of the set ofobservations from the sensor information, estimation of a set of beliefstates, selection of a first value for a first action associated withthe level of automation transparency, and the control of the displaysystem 118 to display the cue based on the selected first value.

The circuitry 202 may include any suitable special-purpose orgeneral-purpose computer, computing entity, or processing deviceincluding various computer hardware or software modules and may beconfigured to execute instructions stored on any applicablecomputer-readable storage media. For example, the circuitry 202 mayinclude a microprocessor, a microcontroller, a digital signal processor(DSP), an application-specific integrated circuit (ASIC), aField-Programmable Gate Array (FPGA), or any other digital or analogcircuitry configured to interpret and/or to execute program instructionsand/or to process data.

Although illustrated as a single circuitry in FIG. 2, the circuitry 202may include any number of processors configured to, individually orcollectively, perform or direct performance of any number of operationsof the autonomous driving agent 102, as described in the presentdisclosure. Additionally, one or more of the processors may be presenton one or more different electronic devices, such as different servers.In some embodiments, the circuitry 202 may be configured to interpretand/or execute program instructions and/or process data stored in thememory 204 and/or a persistent data storage. In some embodiments, thecircuitry 202 may fetch program instructions from a persistent datastorage and workload the program instructions in the memory 204. Afterthe program instructions are loaded into the memory 204, the circuitry202 may execute the program instructions. Some of the examples of thecircuitry 202 may be a Graphical Processing Unit (GPU), a CentralProcessing Unit (CPU), a Reduced Instruction Set Computer (RISC)processor, an ASIC processor, a Complex Instruction Set Computer (CISC)processor, a co-processor, and/or a combination thereof.

The memory 204 may include suitable logic, circuitry, interfaces, and/orcode that may be configured to store the program instructions executableby the circuitry 202. In certain embodiments, the memory 204 may beconfigured to store operating systems and associatedapplication-specific information. Examples of information stored in thememory 204 may include, but are not limited to, the sensor information,such as the scene information and/or the human behavioral data. Thememory 204 may include computer-readable storage media for carrying orhaving computer-executable instructions or data structures storedthereon. Such computer-readable storage media may include any availablemedia that may be accessed by a general-purpose or a special-purposecomputer, such as the circuitry 202. By way of example, and notlimitation, such computer-readable storage media may include tangible ornon-transitory computer-readable storage media including Random AccessMemory (RAM), Read-Only Memory (ROM), Electrically Erasable ProgrammableRead-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM), HardDisk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and/or aSecure Digital (SD) card or other optical disk storage, magnetic diskstorage or other magnetic storage devices, flash memory devices (e.g.,solid state memory devices), or any other storage medium which may beused to carry or store particular program code in the form ofcomputer-executable instructions or data structures and which may beaccessed by a general-purpose or special-purpose computer. Combinationsof the above may also be included within the scope of computer-readablestorage media. Computer-executable instructions may include, forexample, instructions and data configured to cause the circuitry 202 toperform a certain operation or a group of operations associated with theautonomous driving agent 102.

The I/O interface 206 may include suitable logic, circuitry, andinterfaces that may be configured to receive a user input and provide anoutput based on the received input. The I/O interface 206 which includesvarious input and output devices, may be configured to communicate withthe circuitry 202. Examples of the I/O interface 206 may include, butare not limited to, a touch screen, a keyboard, a mouse, a joystick, amicrophone, a display (such as the display system 118), and a speaker.

The network interface 208 may include suitable logic, circuitry,interfaces, and/or code that may enable communication among theautonomous driving agent 102 and other external devices, such as thevehicle 104 and the server 108, via the communication network 110. Thenetwork interface 208 may implement known technologies to support wiredand/or wireless communication via the communication network 110. Thenetwork interface 208 may include, but is not limited to, an antenna, afrequency modulation (FM) transceiver, a radio frequency (RF)transceiver, one or more amplifiers, a tuner, one or more oscillators, adigital signal processor, a coder-decoder (CODEC) chipset, a subscriberidentity module (SIM) card, and/or a local buffer.

The network interface 208 may communicate via wired and/or wirelesscommunication with networks, such as the Internet, an Intranet and/or awireless network, such as a cellular telephone network, a wireless localarea network (LAN) and/or a metropolitan area network (MAN). Thecommunication may use any of a plurality of communication standards,protocols and technologies, such as Long Term Evolution (LTE), GlobalSystem for Mobile Communications (GSM), Enhanced Data GSM Environment(EDGE), wideband code division multiple access (W-CDMA), code divisionmultiple access (CDMA), time division multiple access (TDMA), Bluetooth,Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE802.11g and/or IEEE 802.11n), voice over Internet Protocol (VoIP),Wi-MAX, a protocol for email, instant messaging, and/or Short MessageService (SMS).

The functions or operations executed by the autonomous driving agent102, as described in FIG. 1, may be performed by the circuitry 202.Operations executed by the circuitry 202 are described in detail, forexample, in FIG. 6.

FIG. 3 is a block diagram of an exemplary vehicle that implements theautonomous driving agent of FIG. 1, in accordance with an embodiment ofthe disclosure. FIG. 3 is explained in conjunction with elements fromFIG. 1. With reference to FIG. 3, there is shown a block diagram 300 ofthe vehicle 104. The block diagram 300 of the vehicle 104 may includethe autonomous driving agent 102, which may be implemented as part of anIn-vehicle Infotainment (IVI) system or as an ECU (which may include atleast a microprocessor and/or a memory). The vehicle 104 may furtherinclude circuitry 302 as part of the autonomous driving agent 102, thesensor system 106 (which includes the gaze detector 106 a, the frontfacing camera 106 b, and the event logger 106 c), an in-vehicle displaydevice 304 (as part of the display system 118), and a memory 306communicatively coupled to the circuitry 302. One or more userinterfaces (UIs), such as a UI 304 a may be rendered on the in-vehicledisplay device 304.

The circuitry 302 may communicate with the sensor system 106, via anin-vehicle network 308. The vehicle 104 may further include a networkinterface 310 that may connect the vehicle 104 to other externaldevices, such as the server 108, via the communication network 110. Aperson of ordinary skilled in the art will understand that the vehicle104 may also include other suitable components or systems, in additionto the components or systems illustrated herein to describe and explainthe function and operation of the present disclosure. A description ofsuch components or systems is omitted herein for the sake of brevity.

The circuitry 302 may include suitable logic, circuitry, and/orinterfaces that may be configured to execute program instructionsassociated with different operations to be executed by the autonomousdriving agent 102. The circuitry 302 may include any suitablespecial-purpose or general-purpose computer, computing entity, orprocessing device including various computer hardware or softwaremodules and may be configured to execute instructions stored on anyapplicable computer-readable storage media. For example, the circuitry302 may include a microprocessor, a microcontroller, a DSP, an ASIC, aFPGA, or any other digital or analog circuitry configured to interpretand/or to execute program instructions and/or to process data.

Although illustrated as a single circuitry in FIG. 3, the circuitry 302may include any number of processors configured to, individually orcollectively, perform or direct performance of any number of operationsof the autonomous driving agent 102, as described in the presentdisclosure. Additionally, one or more of the processors may be presenton one or more different electronic devices, such as different servers.In some embodiments, the circuitry 302 may be configured to interpretand/or execute program instructions and/or process data stored in thememory 306 and/or a persistent data storage. In some embodiments, thecircuitry 302 may fetch program instructions from a persistent datastorage and workload the program instructions in the memory 306. Afterthe program instructions are loaded into the memory 306, the circuitry302 may execute the program instructions. Some of the examples of thecircuitry 302 may be a GPU, a CPU, a RISC processor, an ASIC processor,a CISC processor, a co-processor, and/or a combination thereof.

The in-vehicle display device 304 may include suitable logic, circuitry,interfaces, and/or code that may be configured to render various typesof information and/or viewable content via the UI 304 a. The UI 304 amay be a customizable or a non-customizable Graphical UI that maydisplay various types of information related to the autonomous drivingagent 102. For example, the UI 304 a may display a visual representationof the surrounding environment 114, such as images or a video of a sceneof the surrounding environment 114 and a cue overlaid over object(s)detected in the visual representation. In an embodiment, the cue maycorrespond to an Augmented-Reality (AR) cue, such as AR bounding boxeson the object(s), or color indicators to classify the object(s).Examples of the in-vehicle display device 304 may include, but are notlimited to, a display of the infotainment head unit, a projection-baseddisplay, a see-through display, and/or an electro-chromic display. In anembodiment, the in-vehicle display device 304 may be implemented as oneof, but not limited to, MID, an automotive HUD, or an instrumentcluster.

The memory 306 may include suitable logic, circuitry, interfaces, and/orcode that may be configured to store the program instructions executableby the circuitry 302. In certain embodiments, the memory 306 may beconfigured to store operating systems and associatedapplication-specific information. Examples of information stored in thememory 306 may include, but are not limited to, the sensor information(such as, the plurality of image frames from the front facing camera 106b, the scene information, information that includes event logsassociated with the human inputs of the human user 120), and the humanattention information associated with human user 120. The functions ofthe memory 306 may be same as the functions of the memory 204 described,for example, in FIG. 2. Therefore, further description of the memory 306is omitted from the disclosure for the sake of brevity.

The in-vehicle network 308 may include a medium through which thevarious control units, components, and/or systems of the vehicle 104 maycommunicate with each other. In accordance with an embodiment,in-vehicle communication of audio/video data may occur by use of MediaOriented Systems Transport (MOST) multimedia network protocol of thein-vehicle network 308 or other suitable network protocols for vehiclecommunication. The MOST-based network may be a separate network from thecontroller area network (CAN). In accordance with an embodiment, theMOST-based network, the CAN, and other in-vehicle networks may co-existin the vehicle 104. The in-vehicle network 308 may facilitate accesscontrol and/or communication among the circuitry 302 of the autonomousdriving agent 102, the sensor system 106, the network interface 310,OBUs, and other ECUs, such as Engine Control Module (ECM) or atelematics control unit (TCU) of the vehicle 104.

Various devices or components in the vehicle 104 may connect to thein-vehicle network 308, in accordance with various wired and wirelesscommunication protocols. Examples of the wired and wirelesscommunication protocols for the in-vehicle network 308 may include, butare not limited to, a vehicle area network (VAN), a CAN bus, DomesticDigital Bus (D2B), Time-Triggered Protocol (TTP), FlexRay, IEEE 1394,Carrier Sense Multiple Access With Collision Detection (CSMA/CD) baseddata communication protocol, Inter-Integrated Circuit (I²C), InterEquipment Bus (IEBus), Society of Automotive Engineers (SAE) J1708, SAEJ1939, International Organization for Standardization (ISO) 11992, ISO11783, Media Oriented Systems Transport (MOST), MOST25, MOST50, MOST150,Plastic optical fiber (POF), Power-line communication (PLC), SerialPeripheral Interface (SPI) bus, and/or Local Interconnect Network (LIN).

The network interface 310 may include suitable logic, circuitry,interfaces, and/or code that may enable communication among the vehicle104 and other external devices, such as the server 108, via thecommunication network 110. The network interface 310 may implement knowntechnologies to support wired and/or wireless communication via thecommunication network 110. The network interface 208 may include, but isnot limited to, an antenna, a frequency modulation (FM) transceiver, aradio frequency (RF) transceiver, one or more amplifiers, a tuner, oneor more oscillators, a digital signal processor, a coder-decoder (CODEC)chipset, a subscriber identity module (SIM) card, and/or a local buffer.The functions of network interface 310 may be same as the functions ofthe network interface 208 described, for example, in FIG. 2. Therefore,further description of the network interface 310 is omitted from thedisclosure for the sake of brevity.

It should be noted that some or all of the functions and/or operationsperformed by the circuitry 202 (as described in FIG. 2) may be performedby the circuitry 302, without a deviation from the scope of thedisclosure.

FIG. 4 is a diagram that illustrates an exemplary trust-workload modelfor calibration of human trust on the autonomous driving agent of FIG.1, in accordance with an embodiment of the disclosure. FIG. 4 isexplained in conjunction with elements from FIG. 1, 2, or 3. Withreference to FIG. 4, there is shown a diagram 400 of a trainedtrust-workload model 402. The trained trust-workload model 402 mayinclude a first action 404 a, a second action 404 b, and a third action404 c (collectively referred to as a set of actions 404). The trainedtrust-workload model 402 may further include a first state 406 a and asecond state 406 b (collectively referred to as a set of states 406).Further, the trained trust-workload model 402 may include a firstobservation 408 a and a second observation 408 b (collectively referredto as a set of observations 408).

The trained trust-workload model 402 may be a POMDP model, where each ofthe set of states 406 of the trained trust-workload model 402 may bemeasured by a belief state. Therefore, a set of belief states associatedwith the set of states 406 may include a first belief state for thefirst state 406 a and a second belief state for the second state 406 b.Herein, the first state 406 a may be for human trust on the autonomousdriving agent 102 and the second state 406 b may be for human'scognitive workload (also referred to as workload) in the course ofjourney. The first belief state may include a first probabilisticmeasure to observe the first state 406 a of the human trust. Similarly,the second belief state may include a second probabilistic measure toobserve the second state 406 b of the human's cognitive workload.

Interaction with the autonomous driving agent 102 (e.g., in a level 2driving automation) may be characterized by human's reliance, or lackthereof, on the autonomous driving agent 102. There may be an associatedeye-gaze behavior (as part of the human attention information) which maycorrespond to the human's supervision of the autonomous driving agent102 when in the course of journey. It may be assumed thatcharacteristics of human behavior, i.e., human's reliance on theautonomous driving agent 102 or attention (measured by gaze behavior)may be dependent on the human trust and the workload (i.e. also referredto as the human's cognitive workload). Furthermore, the human trust andthe workload may be influenced by characteristics of the automationtransparency, the automation reliability as well as the scene complexity(as measured from the surrounding environment 114). It may be furtherassumed that dynamics of the human trust and the workload may follow theMarkov property, and therefore, human trust-workload behavior may bemodelled as a POMDP model (such as the trained trust-workload model402).

As the human trust and the workload cannot be directly observed, thetrained trust-workload model 402 may define the set of states 406consisting of tuples of the first state 406 a of the human trust and thesecond state 406 b of the human's cognitive workload. The first state406 a can either be Low Trust or High Trust. Similarly, the second state406 b of the human's cognitive workload can either be Low Workload orHigh Workload. As the set of states 406 may be influenced bycharacteristics of the autonomous driving agent 102 and the surroundingenvironment 114, the trained trust-workload model 402 may define the setof actions 404 consisting of tuples of the first action 404 a associatedwith the automation transparency, the second action 404 b associatedwith the automation reliability, and the third action 404 c associatedwith the scene complexity. Observable characteristics of the human user120 may be defined as the set of observations 408 consisting of thefirst observation 408 a associated with the human reliance on theautonomous driving agent 102 and the second observation 408 b associatedwith the human's attention information (includes gaze position).

The set of actions 404 may include a set of controllable actions and aset of uncontrollable actions. The set of controllable actions mayinclude the first action 404 a that may be associated with a level ofthe automation transparency between the human user 120 of the vehicle104 and the autonomous driving agent 102. For example, the first action404 a may correspond to a decision to display a type of information bythe autonomous driving agent 102 in the form of cues via the displaysystem 118 in the course of the journey.

The set of uncontrollable actions may include the second action 404 band the third action 404 c. The second action 404 b may be associatedwith the automation reliability on the autonomous driving agent 102 inthe course of the journey. The automation reliability may be defined atan intersection in terms of the distance up to which the autonomousdriving agent 102 may stop the vehicle 104 before a stop line. Theautomation reliability may be defined to be low (reliance low) if thevehicle 104 stops before a first threshold distance (such as 5 meters)from the stop line or crosses the stop line. The reliability may bedefined to be medium (reliance medium) if the vehicle 104 stops betweenthe first threshold distance (such as 5 meters) and a second thresholddistance (such as 15 meters) before the stop line. The reliability maybe defined to be high (reliance high) if the vehicle 104 stops more thanthe second threshold distance (such as 15 meters) away from the stopline. Such a reliability definition may be similar to drivingaggressiveness, which affects the perceived trustworthiness of theautonomous driving agent 102.

The third action 404 c associated with the autonomous driving agent 102may be defined as the scene complexity. The scene complexity may becharacterized by both traffic density (traffic low or traffic high) andintersection complexity (pedestrians absent or pedestrians present). Thescene complexity may be determined from the scene information (forexample, images acquired from the front facing camera 106 b). Forexample, a low complexity scene may include a vehicle-only or apedestrian-only scenario on a road intersection. Whereas a highcomplexity scene may include a combination of one or more vehicles andone or more pedestrians on the road intersection.

The first observation 408 a may be associated with a level of humanreliance on the autonomous driving agent 102. The level of humanreliance may correspond to one of: a human takeover of controls of thevehicle 104 by the human user 120 or a takeover of the controls of thevehicle 104 by the autonomous driving agent 102. Similarly, the secondobservation 408 b may be associated with the human attention informationassociated with a scene of the surrounding environment 114 of thevehicle 104. For example, the human attention information may includedatapoints, such as a human gaze on an object of a certain type in thescene of the surrounding environment. The position of the human gaze maybe classified as belonging to one of: road, vehicle, pedestrian, asidewalk, or others in each image frame collected by the gaze detector106 a and/or the front facing camera 106 b.

The first state 406 a of the human trust and the second state 406 b ofthe human's cognitive workload may be coupled and may influence the useof the autonomous driving agent 102. Modeling the first state 406 a andthe second state 406 b as coupled states at a given time presentsseveral challenges. Among these challenges is the difficulty ofexplicitly distinguishing individual trust and workload states aftermodel parameterization. In order to overcome this, it may be assumedthat at any given time, the human trust and the human's cognitiveworkload are independent but that the human trust and the human'scognitive workload at current time affect a next state of the humantrust as well as a next state of the human's cognitive workload. In thisway, the trained trust-workload model 402 may capture dynamics betweenthe human trust and the human's cognitive workload as they evolve overtime. Furthermore, it may be assumed that the human trust only affectsreliance and the human's cognitive workload only affects the humanattention, such as the gaze position. This may enable the first state406 a of the human trust and the second state 406 b of the human'scognitive workload to be identified based respective emissionprobabilities. These assumptions significantly reduce a number of modelparameters and in turn, the amount of data needed to estimate them. Theyalso result in separate transition probability functions for the humantrust and the human's cognitive workload, as well as independentemission probability functions for reliance, and human attention (suchas in terms of gaze position).

The first action 404 a (associated with the level of automationtransparency) and the second action 404 b (associated with theautomation reliability) may affect the first state 406 a of the humantrust. The first observation 408 a associated with the level of humanreliance on the autonomous driving agent 102 may indicate an effect ofthe first action 404 a and the second action 404 b on the first state406 a (measured by the first belief state) of the human trust on theautonomous driving agent 102 of the vehicle 104. Also, the first action404 a associated with the level of automation transparency, the secondaction 404 b associated with the automation reliability, and the thirdaction 404 c associated with the scene complexity may affect the secondstate 406 b of the human's cognitive workload in the course of thejourney. The second observation 408 b may be associated with the humanattention information may indicate the effect of the first action 404 a,the second action 404 b, and the third action 404 c on the second state406 b (measured by the second belief state) of the human's cognitiveworkload in the course of journey.

The autonomous driving agent 102 may use the trained trust-workloadmodel 402 to optimally select a first value for the first action 404 aassociated a level of the automation transparency between the human user120 and the autonomous driving agent 102. Depending on the selectedfirst value, it may be decided whether to display cue(s) on the displaysystem 118 for a current set of belief states and past actionsassociated with the autonomous driving agent 102.

FIG. 5 is a diagram that illustrates exemplary operations for trainingthe exemplary trust-workload model of FIG. 4, in accordance with anembodiment of the disclosure. FIG. 5 is explained in conjunction withelements from FIGS. 1, 2, 3, and 4. With reference to FIG. 5, there isshown a diagram 500 to depict exemplary operations from 502 to 510 fortraining a trust-workload model, which when deployed on the autonomousdriving agent 102, may enable the autonomous driving agent 102 tocalibrate the human trust of the human user 120 on the autonomousdriving agent 102. The exemplary operations illustrated in the diagram500 may start at 502 and may be performed by any computing system,apparatus, or device, such as by the server 108 of FIG. 1. Althoughillustrated with discrete blocks, the operations associated with one ormore of the blocks of the diagram 500 may be divided into additionalblocks, combined into fewer blocks, or eliminated, depending on theparticular implementation.

At 502, data preparation may be performed. For data preparation, theserver 108 may generate a training dataset 512 for an initialtrust-workload model 514. For example, the server 108 may generate thetraining dataset 512 based on information which may include a sequenceof action-observation data for each human test subject as an interactionsequence at each intersection. The sequence of action-observation datamay be acquired based on a visual attention, and human reliance ortake-over information (e.g., vehicle steering/acceleration controltake-over or a human non-intervention) from the respective human subjectwhile the human test subject uses a simulated or real autonomous testdriving system.

For example, the training dataset 512 may include datapoints of 16participants (i.e., human test subjects) for 8 (i.e., 2×2×2) driveconditions each. The 8 drive conditions may include 2 levels of ARpresentations (i.e., annotated and un-annotated), 2 levels of trafficdensity (i.e., low traffic and high traffic), and 2 levels ofintersection complexity (i.e., vehicle only or a combination of vehiclesand pedestrians). Further, the training dataset 512 may includeinformation related to physiological responses of the human testsubjects (e.g., eye-tracking data) and behavioral responses of the humantest subjects (e.g., steering and braking/acceleration takeoverresponses). In addition, the training dataset 512 may include 3 levelsof driving automation reliability (i.e., low, medium, and high).

At 504, trust-workload model may be initialized. For modelinitialization, the server 108 may define a structure of the initialtrust-workload model 514 as a set of nodes. The set of nodes of theinitial trust-workload model 514 may include input nodes, intermediatenodes, and output nodes. As dynamics of the human trust and human'scognitive workload (also referred to as workload) may follow the Markovproperty, the initial trust-workload model 514 may be modelled for thehuman trust-workload behavior as a POMDP model. As a POMDP, the initialtrust-workload model 514 may include a set of actions (i.e., as theinput nodes), a set of states (i.e., as the intermediate nodes), and aset of observations (i.e., as the output nodes). The input nodes maydenote the automation transparency, the automation reliability, thescene complexity (in terms of traffic density and/or intersectioncomplexity) as the set of actions. The intermediate nodes of the initialtrust-workload model 514 may include a first state to represent humantrust on the simulated or real autonomous test driving system and asecond state to represent the human's cognitive workload during a courseof a test journey. The output nodes may denote a level of human relianceon the simulated or real autonomous test driving system and humanattention information (based on the eye-gaze) of the human testsubjects.

The server 108 may define the structure of the initial trust-workloadmodel 514 as an association between the set of nodes. For example, inthe initial trust-workload model 514, each input node (action) may beassociated with each intermediate node (state). Output nodes(observations) may be associated with each respective intermediate node.

By way of example, and not limitation, the inputs of the initialtrust-workload model 514 may be determined as automation transparency(i.e., an AR presentation level), the automation reliability, or thescene complexity. The server 108 may also determine outputs of theinitial trust-workload model 514 as human reliance (i.e., a behavioralresponse) on autonomation and human attention information (e.g.,measured in terms of eye-gaze behavior, i.e., a physiological response)of the human test subjects on object type(s) present in the scenesurrounding a test vehicle. The reliance may be represented by adiscrete value including one of a takeover of vehicle controls (i.e.,simulated vehicle driver controls in this case) by the human testsubject or a takeover of the vehicle controls by the simulated or realautonomous test driving system. Similarly, the eye-gaze may berepresented by an object type on which an eye gaze of the human testsubject may be detected. For example, the object type may include aroad, a sidewalk, a pedestrian, a vehicle, smartphone, in-vehicleobjects, sky, or any other objects (e.g., buildings).

At 506, model parameters may be estimated. Before estimation ofparameters, it may be assumed that the human trust and cognitiveworkload are independent of each other, as given by equation (1), asfollows:P(Trust, Workload)=P(Trust)×P(Workload)  (1)Where,

Trust may represent the state of the human trust on the autonomation;and

Workload may represent the human's cognitive workload during course ofthe journey of the test vehicle.

To estimate parameters of the initial trust-workload model 514 using thetraining dataset 512, an optimization problem may be formulated andsolved to maximize the likelihood of observing sequences of observationsfor given sequences of actions. The Baum-Welch algorithm is typicallyused to address a similar problem for estimating hidden Markov models(HMMs). However, HMMs lack the notion of actions; therefore, forestimating the parameters of the initial trust-workload model 514, amodified version of the Baum-Welch algorithm that accounts for actionsalong with state and observation independence assumptions may be used.

The Baum-Welch technique may include the use of anExpectation-Maximization (EM) optimization technique with aforward-backward algorithm on the training dataset 512 to estimate theparameters of the initial trust-workload model 514. For example, a1000-point multi-start EM optimization may be performed to detect aglobal optima and avoid a local minima of the parameters.

At 608, model simplification may be performed. The server 108 maydetermine an action space associated with possible values of actions.For example, the server 108 may determine an action space cardinalityassociated with the training dataset 512 as 26. The set of possibleactions for such an action space cardinality of 26 is provided in Table1, as follows:

TABLE 1 Set of possible actions associated with the training dataset 512Action Set Details Road_AR-OFF Road without vehicle/pedestrian, AR “OFF”Road_AR-ON Road without vehicle/pedestrian, AR “ON” LT_VO_rel-LOW_AR-OFFLow traffic, vehicle only, low reliability, AR “OFF” LT_VO_rel-LOW_AR-ONLow traffic, vehicle only, low reliability, AR “ON” LT_VO_rel-MED_AR-OFFLow traffic, vehicle only, medium reliability, AR “OFF”LT_VO_rel-MED_AR-ON Low traffic, vehicle only, medium reliability, AR“ON” LT_VO_rel-HIGH_AR-OFF Low traffic, vehicle only, high reliability,AR “OFF” LT_VO_rel-HIGH_AR-ON Low traffic, vehicle only, highreliability, AR “ON” LT_VP_rel-LOW_AR-OFF Low traffic,vehicle/pedestrian, low reliability, AR “OFF” LT_VP_rel-LOW_AR-ON Lowtraffic, vehicle/pedestrian, low reliability, AR “ON”LT_VP_rel-MED_AR-OFF Low traffic, vehicle/pedestrian, mediumreliability, AR “OFF” LT_VP_rel-MED_AR-ON Low traffic,vehicle/pedestrian, medium reliability, AR “ON” LT_VP_rel-HIGH_AR-OFFLow traffic, vehicle/pedestrian, high reliability, AR “OFF”LT_VP_rel-HIGH_AR-ON Low traffic, vehicle/pedestrian, high reliability,AR “ON” HT_VO_rel-LOW_AR-OFF High traffic, vehicle only, lowreliability, AR “OFF” HT_VO_rel-LOW_AR-ON High traffic, vehicle only,low reliability, AR “ON” HT_VO_rel-MED_AR-OFF High traffic, vehicleonly, medium reliability, AR “OFF” HT_VO_rel-MED_AR-ON High traffic,vehicle only, medium reliability, AR “ON” HT_VO_rel-HIGH_AR-OFF Hightraffic, vehicle only, high reliability, AR “OFF” HT_VO_rel-HIGH_AR-ONHigh traffic, vehicle only, high reliability, AR “ON”HT_VP_rel-LOW_AR-OFF High traffic, vehicle/pedestrian, low reliability,AR “OFF” HT_VP_rel-LOW_AR-ON High traffic, vehicle/pedestrian, lowreliability, AR “ON” HT_VP_rel-MED_AR-OFF High traffic,vehicle/pedestrian, medium reliability, AR “OFF” HT_VP_rel-MED_AR-ONHigh traffic, vehicle/pedestrian, medium reliability, AR “ON”HT_VP_rel-HIGH_AR-OFF High traffic, vehicle/pedestrian, highreliability, AR “OFF” HT_VP_rel-HIGH_AR-ON High traffic,vehicle/pedestrian, high reliability, AR “ON”

A person having ordinary skill in the art will understand that the setof possible actions associated with the training dataset 512 (as shownin Table 1) are merely provided as an example and should not beconstrued as limiting for the scope of the disclosure.

Not all actions from the set of possible actions may affect the set ofstates. Therefore, to obtain a final trust-workload model 516 (i.e. thetrained trust-workload model 402 of FIG. 4) with a best generalizabilityfor the training dataset 512, a subset of actions that directly affecttrust and workload dynamics may be determined from the set of possibleactions and a possible trust-workload model with the determined subsetof action for the human trust and the workload may be trained.Similarly, other possible trust-workload models may be trained withdifferent subsets of action for the human trust and the workload. It maybe assumed that the action “automation reliability” may be selected as adefault action that may affect the first state of the human trust.

At 510, model selection may be performed. The server 108 may select atrained trust-workload model as the final trust-workload model 516 (i.e.the trained a trust-workload model 402) from all possible trust-workloadmodels trained at 508. For such selection, the server 108 may perform a3-fold cross validation of each possible trust-workload model (trainedat 508). For example, the server 108 may perform 24 iterations for allthe possible trust-workload models based on a similar distribution ofdata associated with the human test subjects in the training dataset 512and trails across the folds. Multiple iterations may be useful inreduction of the uncertainty associated with the cross validation.

In an embodiment, the server 108 may select the final trust-workloadmodel 516 based on an Akaike Information Criterion (AIC) scorecalculated for each of the possible trust-workload models. The server108 may use the AIC score to test a fitness of each possibletrust-workload model with the training dataset 512 to select a bestfitting model as the final trust-workload model 516. The AIC score forall the possible trust-workload models may be minimized such that thetrust-workload model with the maximum value of the AIC score (or thehighest likelihood) may be selected as the final trust-workload model516 (or the trained trust-workload model 402 of FIG. 4). The server 108may determine the AIC scores for each of the possible trust-workloadmodels using equation (2), given as follows:AIC score=2k−2 log_(e)(L)  (2)Where,k may represent a number of parameters (variables in plus an intercept)of a trust-workload model; andL may represent a likelihood value of the trust-workload model.

The final trust-workload model 516 may include an initial probabilityfor the first state of the human trust (π(S_(T))) and an initialprobability for the second state of the human's cognitive workload(π(S_(W))). The final trust-workload model 516 may further includeemission probability functions ϵ_(T)(O_(R),S_(T)) and ϵ_(W)(O_(G),S_(W))for the level of human reliance on the simulated or real autonomous testdriving system and for the human attention information, respectively.Further, the final trust-workload model 516 may include transitionprobability functions T_(T)(S′_(T)|S_(T), S_(W), a) andT_(W)(S′_(W)|S_(T), S_(W), a) for transition of the first state of thehuman trust and the second state of the human's cognitive workload,respectively, from a current value to a future value.

FIG. 6 is a diagram that illustrates exemplary operations for humantrust calibration for the autonomous driving agent of FIG. 1, inaccordance with an embodiment of the disclosure. FIG. 6 is explained inconjunction with elements from FIGS. 1, 2, 3, and 4. With reference toFIG. 6, there is shown a diagram 600 that depicts exemplary operationsfrom 602 to 610 for calibration of human trust on the autonomous drivingagent 102 of FIG. 1. The exemplary operations illustrated in the diagram600 may start at 602 and may be performed by any computing system,apparatus, or device, such as by the autonomous driving agent 102 ofFIG. 1, FIG. 2, or FIG. 3. In order to calibrate the human trust at alltime-steps in the course of journey, the operations from 602 to 610 maybe iteratively performed for every time-step in the course of thejourney. Also, the trained trust-workload model 402 of FIG. 4 or thefinal trust-workload model 516 of FIG. 5 may be deployed on theautonomous driving agent 102. Although illustrated with discrete blocks,the operations associated with one or more of the blocks of the diagram600 may be divided into additional blocks, combined into fewer blocks,or eliminated, depending on the particular implementation.

At 602, data acquisition may be performed. For the data acquisition, theautonomous driving agent 102 may receive sensor information 612 from thesensor system 106. The sensor information 612 may include sceneinformation 612 a associated with the surrounding environment 114 andhuman behavioral data 612 b associated with the vehicle 104 and thesurrounding environment 114. As an example, the event logger 106 c ofthe sensor system 106 may log events associated with human inputsreceived from the human user 120 to the autonomous driving agent 102.The human inputs may correspond to a takeover of one or more vehiclecontrols of the vehicle 104 by the human user 120. Additionally, theevent logger 106 c may log events where the autonomous driving agent 102takes over the one or more vehicle controls of the vehicle 104 forautonomous driving of the vehicle 104. The event logger 106 c may storethe events associated with the human inputs as part of the humanbehavioral data 612 b in the memory 204 (or the memory 306). As anotherexample, the front facing camera 106 b may capture a plurality of imageframes of a scene of the surrounding environment 114. At the same time,the gaze detector 106 a may capture an image of a face of the human user120. The captured plurality of image frames and the captured image ofthe face may be stored as part of the scene information 612 a in thememory 204 (or the memory 306).

At 604, observations may be extracted. The autonomous driving agent 102may extract a set of observations 614 based on the received sensorinformation 612. The set of observations 614 may include a level ofhuman reliance 614 a (that may correspond to the first observation 408 aof FIG. 4) and human attention information 614 b (that may correspond tothe second observation 408 b of FIG. 4).

By way of example, and not limitation, the level of human reliance 614 amay correspond to one of a takeover of the one or more vehicle controlsof the vehicle 104 by the human user 120 or a takeover of the one ormore vehicle controls of the vehicle 104 by the autonomous driving agent102 for the autonomous driving of the vehicle 104. The autonomousdriving agent 102 may determine the level of human reliance 614 a basedon the human behavioral data 612 b included in the sensor information612 and acquired from the event logger 106 c. For example, the level ofhuman reliance 614 a may be determined to have a “HIGH” value in casethe human user 120 does not take over any control of the vehicle 104,while the vehicle 104 is autonomously controlled by the autonomousdriving agent 102. Alternatively, the level of human reliance 614 a maybe determined to have a “LOW” value in case the human user 120 takesover control of the vehicle 104 at multiple instances while the vehicle104 is autonomously controlled by the autonomous driving agent 102. Suchinstances may include the human user 120 taking over one or more of asteering control and/or a braking/acceleration control of the vehicle104 in the course of the journey.

By way of example, and not limitation, the human attention information614 b may include a human gaze on an object type present in a scene ofthe surrounding environment 114. Examples of the object may include, butare not limited to, a vehicle, a pedestrian, a side-walk, a roadportion, or any other type of objects (such as a building, smartphone,sky, or other objects inside or outside of the vehicle 104). Theautonomous driving agent 102 may detect the human gaze on the objecttype based on application of Tobii's attention filter on the sceneinformation 612 a included in the sensor information 612.

At 606, belief states may be estimated. The autonomous driving agent 102may estimate a set of belief states which may include a first beliefstate and a second belief state. The first belief state may be for afirst state (e.g., the first state 406 a) of the human trust on theautonomous driving agent 102. The second belief state may be for asecond state (e.g., the second state 406 b) of the human's cognitiveworkload during the course of journey of the vehicle 104. Each of thefirst belief state and the second belief state may be a probabilisticmeasure of the first state and the second state, respectively, asmodeled by the trained trust-workload model 402.

In an embodiment, the autonomous driving agent 102 may estimate the setof belief states based on the set of observations 614. The estimation ofthe set of belief states may be further based on a set of previousbelief states 616 of the autonomous driving agent 102 and a set ofprevious actions 618 of the autonomous driving agent 102 associated withthe set of previous belief states 616. For the first state and thesecond state, the set of belief states may be estimated by solving aconditional probability distribution function in which the set ofprevious belief states 616, the set of previous actions 618, and the setof observations 614 may be input variables. For example, the estimationof the set of belief states may be based on a conditional probabilitydistribution function, which is given by equation (3), as follows:

$\begin{matrix}{{b( s^{\prime} )} = {{P( { s^{\prime} \middle| o ,a,{b(s)}} )} = \frac{{P( {o,s^{\prime},a} )} \times {\sum\limits_{s \in S}{{P( { s^{\prime} \middle| s ,a} )} \times {b(s)}}}}{\sum\limits_{s \in S}{{P( { o \middle| s^{\prime} ,a} )} \times {\sum\limits_{s \in S}{{P( { s^{\prime} \middle| s ,a} )} \times {b(s)}}}}}}} & (3)\end{matrix}$Where,S may represent a set of all states;s may represent a (previous) state from the set of states;b(s) may represent a previous belief state for the state “s”;s′ may represent a current state;b(s′) may represent a current belief state for the state “s”;o may represent an observation;a may denote an action; andP(.) may denote a probability function.

At 608, value selection may be performed. The autonomous driving agent102 may select, from a set of values, a first value for a first action(e.g., the first action 404 a) associated with the level of automationtransparency between the human user 120 and the autonomous driving agent102. By way of example, and not limitation, the set of values may be fortwo levels of the automation transparency and may include an ON valueand an OFF value. While the ON value may correspond to a first settingof the display system 118 to display a cue, the OFF value may correspondto a second setting of the display system 118 to hide the cue. The cuemay include visual markers for objects which may be present in a sceneof the surrounding environment 114. The cue may indicate an extent bywhich the autonomous driving agent 102 intends to perform, performs,plans actions, reasons, or understands the scene complexity of thesurrounding environment 114. For example, the cue may include boundingboxes or text labels around objects on road or in vicinity of the roadin images captured by the front facing camera.

By way of another example, and not limitation, the first value may beselected from a set of values (τ), which may be represented by (4), asfollows:τ∈{AR ON, AR OFF}  (4)Where,

τ may represent the set of values for the first action associated withthe level of automation transparency;

AR ON may represent a display setting for the display system 118 torender AR cue(s) or annotation(s); and

AR OFF may represent a display setting for the display system 118 tohide or disable AR cue(s) or annotation(s).

Although, the present disclosure discusses two levels of the automationtransparency; however, the present disclosure may be applicable to morethan two levels of the automation transparency with multiple modalitiesof presentation, such as three levels of transparency based on theSituational Awareness Transparency (SAT) framework. Additional factorsthat influence the scene complexity and the automation reliability, suchas weather conditions or lane keeping behavior may also be considered todecide the multiple modalities of the presentation.

The final trust-workload model 516 (or the trained trust-workload model402) may provide the ability to measure trust and workload levels of thehuman user 120 continuously, and in real or near real time, using beliefstate estimates. In order to calibrate the human trust, a rewardfunction may be defined as a function of the first state of the humantrust and the automation reliability. A penalty (e.g., −1) may beallotted when the final trust-workload model 516 predicts that the humanuser 120 is in a state of high trust, given a low automation reliabilityor when it predicts that the human user 120 is in a state of low trust,given a high automation reliability. A reward (e.g., +1) may be allottedwhen the final trust-workload model 516 predicts that the human user 120is in a state of high trust, given the high automation reliability andwhen it predicts the human user 120 is in a state of low trust, giventhe low automation reliability. A discount factor may be selected suchthat the reward after one second (1 seconds) may carry a weight of e⁻¹,given 25 time steps per second. With the reward function and thediscount factor, a control policy may be applied to select the firstvalue for the first action associated with the level of automationtransparency.

In at least one embodiment, the first value for the first action may beselected from the set of values based on a maximization of the rewardfunction. The reward function may be included in the control policy forthe human trust calibration. For example, the control policy for thehuman trust calibration may be defined for the first state (e.g., thefirst state 406 a) of the human trust and the second action (e.g., thesecond action 404 b) associated with the level of automationreliability. In such a case, the objective of the control policy may beto avoid a low human trust on the autonomous driving agent 102 during ahigh automation reliability and avoid a high human trust on theautonomous driving agent 102 during a low automation reliability.Exemplary values (1.0, −1, 0) of the reward function are presented inTable 2, as follows:

TABLE 2 Reward and penalty values Low Medium High Automation AutomationAutomation Reliability Reliability Reliability Low Human Trust 1.0 0.0−1.0 High Human Trust −1.0 0.0 1.0

These exemplary values are for different combinations of values of thefirst state of the human trust (i.e. Low Human Trust and for High HumanTrust) and different values of the level of automation reliability (I.e.Low Automation Reliability, Medium Automation Reliability, and HighAutomation Reliability). A person having ordinary skill in the art willunderstand that above values of the reward function (as shown in Table2) are merely provided as an example and should not be construed aslimiting for the disclosure.

In at least one embodiment, the autonomous driving agent 102 may useQ-MDP method to obtain a near-optimal solution for the selection of thefirst value. In order to account for the set of uncontrollable actions(i.e. the automation reliability and the scene complexity, as alsodiscussed in FIG. 4) that cannot be explicitly changed by the autonomousdriving agent 102, an expected Q-function may be solved by consideringprobabilities of the set of uncontrollable actions

In an embodiment, the autonomous driving agent 102 may use the QMDPmethod to select the first value of the first action. The QMDP methodmay be denoted by (5), as follows:Q _(MDP) :S×A→R  (5)Where,S may represent the set of states;A may represent set of actions; andR may be reward values of the reward function.Using (5), an objective function may be formulated, as given by anequation (6), as follows:

$\begin{matrix}{a^{\star} = {\arg\;{\max_{a}{\sum\limits_{s \in S}{{b(s)} \times {Q_{MDP}( {s,a} )}}}}}} & (6)\end{matrix}$where,a* may represent the selected first value of the first action a;S may represent the set of states; andb(s) may represent a current belief state that may be determined basedon b(s′) of equation (3). The autonomous driving agent 102 may selectthe first value of the first action by solving the equation (6).

In at least one embodiment, the autonomous driving agent 102 may solvethe expected Q-function for the set of uncontrollable actions. Forexample, the autonomous driving agent 102 may determine a near-optimalvalue for the level of automation transparency by solving the expectedQ-function (Q^(T)) of a form, which may be represented by (7), asfollows:Q ^(T) :S×τ→R  (7)In order to solve for the Q-MDP function of equation (3), a set ofequations represented by (8), (9), and (10) may be iteratively solved.The set of equations are given as follows:

$\begin{matrix}{{Q_{MDP}( {s,a} )} = {\sum\limits_{s^{\prime} \in S}{{T( { s^{\prime} \middle| s ,a} )} \times ( {{R( { s^{\prime} \middle| s ,a} )} + {\gamma \times {V( s^{\prime} )}}} )}}} & (8) \\{{Q^{T}( {s,T} )} = {\sum\limits_{a_{u} \in A_{u}}{{P( a_{u} )} \times {Q_{MDP}( {s,{a = \lbrack {a_{u},T} \rbrack}} )}}}} & (8) \\{{V(s)} = {\max_{T}{Q^{T}( {s,T} )}}} & (10)\end{matrix}$Where,s′ may represent the current state;s may represent the previous state;S may represent the set of states;au may represent an uncontrollable action a;Au may represent the set of all uncontrollable actions;T(.) may represent a transition probability function for transition of astate from the previous state (s) to current state (s′), given an actiona;R(.) may represent a reward function;V(.) may represent a value function to discount a current state; andγ may represent an experimentally determined discount factor to discountthe reward function.

Based on equations (6), (8), (9), and (10), the autonomous driving agent102 may be able to select the first value (e.g., whether to display thecue or not) as the near-optimal value for the level of automationtransparency, which may be calculated using an equation (11), asfollows:

$\begin{matrix}{T^{\star} = {\arg\;{\max_{T}{\sum\limits_{s \in S}{{b(s)} \times {Q_{MDP}( {s,{a = \lbrack {a_{u},T} \rbrack}} )}}}}}} & (11)\end{matrix}$

At 610, human trust calibration may be performed. For the human trustcalibration, the autonomous driving agent 102 may control the displaysystem 118 associated with the autonomous driving agent 102 to display acue based on the selected first value. For example, the autonomousdriving agent 102 may control the display system 118 to render a view ofthe scene of the surrounding environment 114. The autonomous drivingagent 102 may determine a set of objects that may be visible in therendered view. Such determination may be based on application of objectdetection techniques. Details of the object detection techniques may beknown to one skilled in the art, and therefore, a detailed descriptionfor implementation of the object detection techniques is omitted fromthe disclosure for the sake of brevity. Once determined, the autonomousdriving agent 102 may overlay the cue that may include visual markersover the determined ser of objects. An example presentation of cues isprovided in FIGS. 7A, 7B, and 7C.

It should be noted that even though the reward function is defined interms of the first state of the human trust, the control policy may alsobe dependent on the second state of the human's cognitive workload. Thismay be due to a coupled modeling of the human trust and the human'scognitive workload. As one example, for low automation reliability, thecontrol policy may adopt the presence of cues (ON value or AR_(ON) asthe selected first value) for the automation transparency. The presenceof the cues may allow the human user 120 to make an informed decisionand avoid mistrust on the autonomous driving agent 102. As anotherexample, for medium automation reliability, the control policy may adopthigh transparency (ON value or AR_(ON)) when both the first state of thehuman trust and the second state of the human's cognitive workload arelow. Providing the high automation transparency at low human trust mayhelp to increase the human trust on the autonomous driving agent 102,but it may be avoided when the human's cognitive workload is high. Asanother example, for high automation reliability, the high automationtransparency may only be used when the human's cognitive workload islow. In at least one embodiment, the high automation transparency may beadopted even when the human's cognitive workload is high and whenpedestrians are present. One potential reason for this may be that thepresence of the pedestrians may be interpreted as “higher risk” to thehuman user 120, thereby leading to less trust in the autonomous drivingagent 102 if cues are absent.

In at least one embodiment, the autonomous driving agent 102 may set alevel of detail that may be provided via the cues to the human user 120for a given view of the scene and given human trust-workload dynamics.Examples of such cues may include, but are not limited to, boundingboxes, labels, annotated object(s) in the view, annotated criticalobjects that may have an impact on driving decisions of the autonomousdriving agent 102, annotated objects that may have the attention of thehuman user 120 in the scene, markers, or overlay graphics for a set ofdecisions (e.g., a left turn, a right turn, or a straight movement) orpaths (a lane marking, a stop symbol when approaching a traffic light).

FIG. 7A is a diagram that illustrates an exemplary scenario forcalibration of human trust on the autonomous driving agent of FIG. 1 incourse of a journey, in accordance with an embodiment of the disclosure.FIG. 7B is explained in conjunction with elements from FIGS. 1, 2, 3, 4,5, and 6. With reference to FIG. 7A, there is shown an exemplaryscenario 700 a. In the exemplary scenario 700 a, there is shown a UI 702a displayed on the display system 118. The autonomous driving agent 102may display a visual representation 704 a of a driving scene 706 a ontothe UI 702 a displayed on the display system 118. The driving scene 706a may be of the surrounding environment of the vehicle 104 and may beacquired by the front facing camera 106 b. For this exemplary scenario700 a, the first state of the human trust on the autonomous drivingagent 102 may be high and the human's cognitive workload may be low,while the automation reliability may be high. In such a case, the firstvalue may be determined as “AR OFF” for the first action associated withthe level of automation transparency between the human user 120 and theautonomous driving agent 102. The autonomous driving agent 102 maycontrol the display system 118 to disable or hide an overlay of AR cuesover objects, which may be detected in the visual representation 704 a.

FIG. 7B is a diagram that illustrates an exemplary scenario forcalibration of human trust on the autonomous driving agent of FIG. 1 incourse of a journey, in accordance with an embodiment of the disclosure.FIG. 7B is explained in conjunction with elements from FIGS. 1, 2, 3, 4,5, and 6. With reference to FIG. 7B, there is shown an exemplaryscenario 700 b. In the exemplary scenario 700 b, there is shown a UI 702b displayed on the display system 118. The autonomous driving agent 102may display a visual representation 704 b of a driving scene 706 b ontothe UI 702 b displayed on the display system 118. The driving scene 706b may be of the surrounding environment of the vehicle 104 and may beacquired by the front facing camera 106 b. For this exemplary scenario700 b, the first state of the human trust on the autonomous drivingagent 102 may be low and the human's cognitive workload may be low,while the automation reliability may be low. Also, the eye gaze of thehuman user 120 may be set on a pedestrian 708 who may be crossing theroad, as shown in the visual representation 704 b. In such a case, thefirst value may be determined as “AR ON” for the first action associatedwith the level of automation transparency between the human user 120 andthe autonomous driving agent 102. The autonomous driving agent 102 maycontrol the display system 118 to display an AR cue 710 over thepedestrian 708, which may be detected in the visual representation 704b. Herein, the AR cue 710 is show as a bounding box, which may enclose aregion that includes the pedestrian 708.

FIG. 7C is a diagram that illustrates an exemplary scenario forcalibration of human trust on the autonomous driving agent of FIG. 1 incourse of a journey, in accordance with an embodiment of the disclosure.FIG. 7C is explained in conjunction with elements from FIGS. 1, 2, 3, 4,5, and 6. With reference to FIG. 7C, there is shown an exemplaryscenario 700 c. In the exemplary scenario 700 c, there is shown a UI 702c displayed on the display system 118. The autonomous driving agent 102may display a visual representation 704 c of a driving scene 706 c ontothe UI 702 c displayed on the display system 118. The driving scene 706c may be of the surrounding environment of the vehicle 104 and may beacquired by the front facing camera 106 b. For this exemplary scenario700 b, the first state of the human trust on the autonomous drivingagent 102 may be low and the human's cognitive workload may be high,while the automation reliability may be low. Also, the eye gaze of thehuman user 120 may be set on road 712. The scene complexity associatedwith the driving scene 706 c may be high with respect to that associatedwith the driving scene 706 b. In such a case, the first value may bedetermined as “AR_(ON)” for the first action associated with the levelof automation transparency. The autonomous driving agent 102 may controlthe display system 118 to display multiple AR cues over differentobjects, which may be detected in the visual representation 704 c. Inthis case, the multiple AR cues may include a bounding box 714 aroundanother vehicle 716 in FOV of the vehicle 104 and an arrow graphic 718overlaid on the road 712 indicating a projected moving direction of thevehicle 104 in the driving scene 706 c.

It should be noted that that the UI 702 a, the UI 702 b, and the UI 702c merely present examples of AR cues for a limited set of drivingscenes. The disclosure may not be limited to these limited set ofdriving scenes. In at least one embodiment, other types of AR cues mayalso be displayed on the display system for various types of drivingscenes, without a departure from the scope of the disclosure.

FIG. 8 is a flowchart that illustrates an exemplary method forcalibration of human trust on the autonomous driving agent of FIG. 1, inaccordance with an embodiment of the disclosure. FIG. 8 is explained inconjunction with elements from FIGS. 1, 2, 3, 4, 5, 6, 7A, 7B, and 7C.With reference to FIG. 8, there is shown a flowchart 800. The methodillustrated in the flowchart 800 may start at 802 and proceed to 804.The method illustrated in the flowchart 800 may be performed by anycomputing system, apparatus, or device, such as by the autonomousdriving agent 102.

At 804, the set of observations 614 which include the level of humanreliance 614 a and the human attention information 614 b may bedetermined from the sensor information 612 acquired via the sensorsystem 106 of the vehicle 104. In an embodiment, the circuitry 202 (orthe circuitry 302) may receive the sensor information 612 from thesensor system 106 of the vehicle 104. The sensor information 612 mayinclude the human behavioral data 612 b and the scene information 612 a.The circuitry 202 may determine the set of observations 614 from thesensor information 612. For example, the circuitry 202 may determine thelevel of human reliance 614 a from the human behavioral data 612 b andthe human attention information 614 b from the scene information 612 a.

At 806, based on the determined set of observations 614, a set of beliefstates for the first state of the human trust and the second state ofthe human's cognitive workload may be estimated. In an embodiment, thecircuitry 202 (or the circuitry 302) may estimate the set of beliefstates for the first state of the human trust and the second state ofthe human's cognitive workload based on the determined set ofobservations 614. The estimation of the set of belief states may befurther based on the set of previous actions 618 and the set of previousbelief states 616. Details related to the determination of the set ofbelief states is explained, for example, in FIG. 6.

At 808, based on the estimated set of belief states, a first value of aset of values may be selected for the first action associated with thelevel of automation transparency between the human user 120 and theautonomous driving agent 102. In an embodiment, the circuitry 202 (orthe circuitry 302) may select the first value of the set of values forthe first action associated with the level of automation transparency,based on the estimated set of belief states. The selection of the firstvalue of the set of values for the first action is explained, forexample, in FIG. 6.

At 810, based on the selected first value, the display system 118 may becontrolled to display a cue for the calibration of the human trust onthe autonomous driving agent 102. In an embodiment, the circuitry 202(or the circuitry 302) may control the display system 118 to display thecue for the calibration of the human trust on the autonomous drivingagent 102. Details related to the calibration of the human trust isexplained, for example, in FIGS. 5, 7A, 7B, and 7C. Control may furtherpass to end.

The flowchart 800 is illustrated as discrete operations, such as 804,806, 808, and 810. However, in certain embodiments, such discreteoperations may be further divided into additional operations, combinedinto fewer operations, or eliminated, depending on the particularimplementation without detracting from the essence of the disclosedembodiments.

Various embodiments of the disclosure may provide a non-transitory,computer-readable medium and/or storage medium, and/or a non-transitorymachine readable medium and/or storage medium stored thereon, a set ofinstructions executable by a machine and/or a computer. The set ofinstructions may be executable by the machine and/or the computer of anautonomous driving agent of a vehicle to perform operations that mayinclude determination of a set of observations from sensor informationacquired via the sensor system of the vehicle. The set of observationsmay include human attention information associated with a scene of asurrounding environment of the vehicle and a level of human reliance asindicated by human inputs to the autonomous driving agent. The operationmay further include estimation of a set of belief states for a firststate of human trust on the autonomous driving agent and a second stateof human's cognitive workload in course of a journey, based on thedetermined set of observations. Further, the operations may includeselection of a first value of a set of values for a first actionassociated with a level of automation transparency between a human userof the vehicle and the autonomous driving agent, based on the estimatedset of belief states. The operations may further include control of adisplay system to display a cue for a calibration of the human trust onthe autonomous driving agent, based on the selected first value.

For the purposes of the present disclosure, expressions such as“including”, “comprising”, “incorporating”, “consisting of”, “have”,“is” used to describe and claim the present disclosure are intended tobe construed in a non-exclusive manner, namely allowing for items,components or elements not explicitly described also to be present.Reference to the singular is also to be construed to relate to theplural. Further, all joinder references (e.g., attached, affixed,coupled, connected, and the like) are only used to aid the reader'sunderstanding of the present disclosure, and may not create limitations,particularly as to the position, orientation, or use of the systemsand/or methods disclosed herein. Therefore, joinder references, if any,are to be construed broadly. Moreover, such joinder references do notnecessarily infer that two elements are directly connected to eachother.

The foregoing description of embodiments and examples has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or limiting to the forms described. Numerous modificationsare possible in light of the above teachings. Some of thosemodifications have been discussed and others will be understood by thoseskilled in the art. The embodiments were chosen and described forillustration of various embodiments. The scope is, of course, notlimited to the examples or embodiments set forth herein but can beemployed in any number of applications and equivalent devices by thoseof ordinary skill in the art. Rather it is hereby intended the scope bedefined by the claims appended hereto. Additionally, the features ofvarious implementing embodiments may be combined to form furtherembodiments.

The present disclosure may be realized in hardware, or a combination ofhardware and software. The present disclosure may be realized in acentralized fashion, in at least one computer system, or in adistributed fashion, where different elements may be spread acrossseveral interconnected computer systems. A computer system or otherapparatus adapted for carrying out the methods described herein may besuited. A combination of hardware and software may be a general-purposecomputer system with a computer program that, when loaded and executed,may control the computer system such that it carries out the methodsdescribed herein. The present disclosure may be realized in hardwarethat comprises a portion of an integrated circuit that also performsother functions. It may be understood that, depending on the embodiment,some of the steps described above may be eliminated, while otheradditional steps may be added, and the sequence of steps may be changed.

The present disclosure may also be embedded in a computer programproduct, which comprises all the features that enable the implementationof the methods described herein, and which when loaded in a computersystem is able to carry out these methods. Computer program, in thepresent context, means any expression, in any language, code ornotation, of a set of instructions intended to cause a system with aninformation processing capability to perform a particular functioneither directly, or after either or both of the following: a) conversionto another language, code or notation; b) reproduction in a differentmaterial form. While the present disclosure has been described withreference to certain embodiments, it will be understood by those skilledin the art that various changes may be made, and equivalents may besubstituted without departing from the scope of the present disclosure.In addition, many modifications may be made to adapt a particularsituation or material to the teachings of the present disclosure withoutdeparting from its scope. Therefore, it is intended that the presentdisclosure is not limited to the particular embodiment disclosed, butthat the present disclosure will include all embodiments that fallwithin the scope of the appended claims.

What is claimed is:
 1. An autonomous driving agent for a vehicle,comprising: circuitry coupled to a display system and a sensor system ofthe vehicle, wherein the circuitry is configured to: determine, fromsensor information acquired via the sensor system, a set of observationscomprising human attention information associated with a scene of asurrounding environment of the vehicle and a level of human reliance asindicated by human inputs to the autonomous driving agent; estimate,based on the determined set of observations, a set of belief states fora first state of human trust on the autonomous driving agent and asecond state of a human's cognitive workload during a journey; estimatethe set of belief states further based on a set of previous beliefstates of the autonomous driving agent and a set of previous actions ofthe autonomous driving agent associated with the set of previous beliefstates; estimate the set of belief states for the first state and thesecond state by solving a conditional probability distribution functionfor which the set of previous belief states, the set of previousactions, and the set of observations are input variables; select, basedon the estimated set of belief states, a first value of a set of valuesfor a first action associated with a level of automation transparencybetween a human user of the vehicle and the autonomous driving agent;and based on the selected first value, control the display system todisplay a cue for a calibration of the human trust on the autonomousdriving agent.
 2. The autonomous driving agent according to claim 1,wherein the vehicle is a self-driving vehicle and the autonomous drivingagent is configured to operate the self-driving vehicle based on a levelof automation.
 3. The autonomous driving agent according to claim 1,wherein the first state of the human trust and the second state of thehuman's cognitive workload are states of a Partially Observable MarkovDecision Process (POMDP) model, and the first action is one of a set ofactions of the autonomous driving agent in the POMDP model.
 4. Theautonomous driving agent according to claim 1, wherein the circuitry isfurther configured to receive, from the sensor system of the vehicle,the sensor information comprising scene information associated with thesurrounding environment and human behavioral data associated with thevehicle and the surrounding environment.
 5. The autonomous driving agentaccording to claim 4, wherein the circuitry is configured to: determinea second action associated with a level of automation reliability on theautonomous driving agent during the journey; and determine a thirdaction associated with the autonomous driving agent as a complexity ofroad intersections based on the scene information.
 6. The autonomousdriving agent according to claim 5, wherein the determined first actionand the determined second action affect the first state of the humantrust on the autonomous driving agent, and the level of human relianceon the autonomous driving agent indicates an effect of the determinedfirst action and the determined second action on the first state.
 7. Theautonomous driving agent according to claim 5, wherein the determinedfirst action, the determined second action, and the determined thirdaction affect the second state of the human's cognitive workload, andthe human attention information associated with the scene of thesurrounding environment indicates an effect of the determined firstaction, the determined second action, and the determined third action onthe second state of the human's cognitive workload.
 8. The autonomousdriving agent according to claim 1, wherein the set of belief statescomprises a first belief state for the first state of the human trust onthe autonomous driving agent and a second belief state for the secondstate of the human's cognitive workload, the first belief statecomprises a first probabilistic measure to observe the first state ofthe human trust, and the second belief state comprises a secondprobabilistic measure to observe the second state of the human'scognitive workload.
 9. The autonomous driving agent according to claim1, wherein the level of human reliance on the autonomous driving agentcorresponds to one of: a human takeover of vehicle controls or atakeover of the vehicle controls by the autonomous driving agent. 10.The autonomous driving agent according to claim 1, wherein the circuitryis further configured to detect a human gaze on an object type in thescene of the surrounding environment based on an application of Tobii'sattention filter on scene information of the sensor information, and thehuman attention information comprises the detected human gaze on theobject type.
 11. The autonomous driving agent according to claim 1,wherein the circuitry is further configured to select, from the set ofvalues for the first action, the first value for the first action basedon a maximization of a reward function included in a control policy forthe human trust calibration.
 12. The autonomous driving agent accordingto claim 1, wherein the set of values for the level of automationtransparency comprises an ON value and an OFF value, the ON valuecorresponds to a first setting of the display system to display the cue,and the OFF value corresponds to a second setting of the display systemto hide the cue.
 13. The autonomous driving agent according to claim 1,wherein the circuitry is configured to control the display system to:render a view of the scene of the surrounding environment; determine aset of objects visible in the rendered view; and overlay the cuecomprising visual markers over the determined set of objects.
 14. Theautonomous driving agent according to claim 1, wherein the cue is anAugmented Reality (AR) cue.
 15. The autonomous driving agent accordingto claim 1, wherein the display system is one of: Multi-InformationDisplay (MID), an automotive Head-Up Display (HUD), or an instrumentcluster associated with the vehicle.
 16. A method, comprising: in anautonomous driving agent for a vehicle: determining, from sensorinformation acquired via a sensor system of the vehicle, a set ofobservations comprising human attention information associated with ascene of a surrounding environment of the vehicle and a level of humanreliance as indicated by human inputs to the autonomous driving agent;estimating, based on the determined set of observations, a set of beliefstates for a first state of human trust on the autonomous driving agentand a second state of a human's cognitive workload during a journey;estimating the set of belief states further based on a set of previousbelief states of the autonomous driving agent and a set of previousactions of the autonomous driving agent associated with the set ofprevious belief states; estimating the set of belief states for thefirst state and the second state by solving a conditional probabilitydistribution function for which the set of previous belief states, theset of previous actions, and the set of observations are inputvariables; selecting, based on the estimated set of belief states, afirst value of a set of values for a first action associated with alevel of automation transparency between a human user of the vehicle andthe autonomous driving agent; and based on the determined first value,controlling a display system of the vehicle to display a cue for acalibration of the human trust on the autonomous driving agent.
 17. Anon-transitory computer-readable medium having stored thereon computerimplemented instructions that, when executed by an autonomous drivingagent of a vehicle, causes the autonomous driving agent to executeoperations, the operations comprising: determining, from sensorinformation acquired via a sensor system of the vehicle, a set ofobservations comprising human attention information associated with ascene of a surrounding environment of the vehicle and a level of humanreliance as indicated by human inputs to the autonomous driving agent;estimating, based on the determined set of observations, a set of beliefstates for a first state of human trust on the autonomous driving agentand a second state of a human's cognitive workload during a journey;estimating the set of belief states further based on a set of previousbelief states of the autonomous driving agent and a set of previousactions of the autonomous driving agent associated with the set ofprevious belief states; estimating the set of belief states for thefirst state and the second state by solving a conditional probabilitydistribution function for which the set of previous belief states, theset of previous actions, and the set of observations are inputvariables; selecting, based on the estimated set of belief states, afirst value of a set of values for a first action associated with alevel of automation transparency between a human user of the vehicle andthe autonomous driving agent; and based on the determined first value,controlling a display system of the vehicle to display a cue for acalibration of the human trust on the autonomous driving agent.